Control apparatus, control method, and recording medium

ABSTRACT

A control apparatus configured to control an image-capturing apparatus executes focus adjustment of the image-capturing apparatus in response to detection of a specific marker pattern from an image of an area to be focused by the image-capturing apparatus in an image acquired by image-capturing of the image-capturing apparatus.

BACKGROUND OF THE DISCLOSURE Field

The present disclosure relates to a control technique of controlling animage-capturing apparatus.

Description of the Related Art

In recent years, attention has focused on a technique, in which aplurality of image-capturing apparatuses is installed at differentpositions, the image-capturing apparatuses perform synchronousimage-capturing of a subject, and images from a plurality of viewpointsacquired by the image-capturing are used to generate not only imagescaptured from the installation positions of the image-capturingapparatuses but also an image from a freely-selected viewpoint as avirtual viewpoint image. The generation and viewing of the virtualviewpoint image based on the images from the viewpoints can be achievedas follows, for example. First, the image-capturing apparatuses areinstalled so as to surround the subject, and the images captured by theimage-capturing apparatuses are aggregated in an image-generatingapparatus, such as a server. The image-generating apparatus thenexecutes processing, such as rendering, based on a virtual viewpointusing the images captured by the image-capturing apparatuses to generatethe virtual viewpoint image, and transmits the virtual viewpoint imageto a user's viewing terminal. Accordingly, the user's viewing terminalcan display the virtual viewpoint image. This technique of generatingthe virtual viewpoint image enables creation of visually impactfulcontents from a viewpoint by generating the virtual viewpoint imagedepending on a viewpoint designated by an image content creator fromimages captured in, for example, a succor game and a basketball game.The technique also enables the user viewing the contents to move aviewpoint by himself/herself using a controller, a tablet device, or thelike mounted on the viewing terminal, and enables the image-generatingapparatus to generate the virtual viewpoint image corresponding to theviewpoint. In such a case, the user can watch the game from his/herviewpoint. Service using such a virtual viewpoint image can add realismas if the user were at the scene, as compared to a conventional capturedimage with which a viewpoint cannot be changed. Japanese PatentApplication Laid-Open No. 2014-215828 discusses a technique, in which aplurality of image-capturing apparatuses is installed so as to surrounda gaze point, and a virtual viewpoint image from a virtual viewpointdesignated by the user is generated using images captured by theimage-capturing apparatuses.

In a case where images captured by the image-capturing apparatuses areused to generate the virtual viewpoint image corresponding to a scenethat occurs in the vicinity of the gaze point, the image-capturingapparatuses are to have undergone focus adjustment so as to focus on thevicinity of the gaze point. As a method of the focus adjustment, aconceivable method is to execute the focus adjustment of theimage-capturing apparatuses using an installed marker such that theimage-capturing apparatuses focus on the marker.

However, sufficient time to the focus adjustment may not be allocateddue to, for example, constraints of time available to install the markerdepending on a subject on which the marker is installed. Thus, there isa demand for executing the focus adjustment of each of theimage-capturing apparatuses in a short period of time.

SUMMARY OF THE DISCLOSURE

According to an aspect of the embodiments, a control apparatusconfigured to control an image-capturing apparatus, the controlapparatus includes an identification unit configured to identify an areato be focused by the image-capturing apparatus in an image-capturingtarget area for which the image-capturing apparatus performsimage-capturing, a detection unit configured to detect a specific markerpattern from an image acquired by the image-capturing of theimage-capturing apparatus, and an execution unit configured to executefocus adjustment of the image-capturing apparatus in response to thedetection of the specific marker pattern from an image of the area thatis to be focused by the image-capturing apparatus and has beenidentified in the acquired image.

Further features of the disclosure will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating an arrangement example of a pluralityof image-capturing apparatuses and respective gaze points of theimage-capturing apparatuses.

FIG. 2 is a block diagram illustrating a system configuration example.

FIG. 3 is a block diagram illustrating a configuration example of acontrol unit according to a first exemplary embodiment.

FIG. 4 is a diagram illustrating a relationship between a distance to asubject and an amount of blur.

FIG. 5 is a diagram illustrating an example of focus adjustment.

FIG. 6 is a diagram illustrating another example of focus adjustment.

FIG. 7 is a diagram illustrating an example in which areas to be focusedon is clearly specified on a screen.

FIG. 8 is a diagram illustrating a positional relationship in athree-dimensional (3D) coordinate space between each image-capturingapparatus and a field.

FIG. 9 is a diagram illustrating a positional relationship betweencoordinates on an image captured by an image-capturing apparatus and 3Dcoordinates.

FIGS. 10A to 10C are diagrams each illustrating a marker patternaccording to the first exemplary embodiment.

FIG. 11 is a diagram illustrating an example in which a person holds aboard on which the marker pattern is printed.

FIGS. 12A and 12B are diagrams each illustrating an example of movingthe marker pattern on the field.

FIG. 13 is a flowchart of processing in the control unit according tothe first exemplary embodiment.

FIG. 14 is a flowchart of focus area identification processing.

FIG. 15 is a flowchart illustrating simple position estimationprocessing.

FIG. 16 is a block diagram illustrating a configuration example of acontrol unit according to a second exemplary embodiment.

FIG. 17 is a flowchart of processing in the control unit according tothe second exemplary embodiment.

FIGS. 18A to 18F are diagrams each illustrating a marker patternaccording to the second exemplary embodiment.

FIG. 19 is a block diagram illustrating a hardware configurationexample.

DESCRIPTION OF THE EMBODIMENTS

Exemplary embodiments of the disclosure will be described in detailbelow with reference to the accompanying drawings. Configurations of theexemplary embodiments described below are mere examples, and thedisclosure is not limited to the illustrated configurations.

A first exemplary embodiment of the disclosure will be described below.FIG. 1 is a diagram schematically illustrating an arrangement example ofa plurality of image-capturing apparatuses which are controlled by acontrol apparatus according to the present exemplary embodiment. Asillustrated in FIG. 1, a plurality of image-capturing apparatuses 101 to110 is installed at mutually different positions. The image-capturingapparatuses 101 to 110 perform synchronous image-capturing of a subjector the like. In the case of the example illustrated in FIG. 1, a field130 of a succor stadium is set as an image-capturing target area, andthe plurality of (ten) image-capturing apparatuses 101 to 110 isarranged so as to surround the field 130. The image-capturingapparatuses 101 to 110 are each provided with an autofocus function, andeach capable of remote-controlling a photographing direction, a focallength of a lens, and a focus position.

In the present exemplary embodiment, respective photographing directionsof the image-capturing apparatuses 101 to 110 are adjusted such that theimage-capturing apparatuses 101 to 110 respectively are pointed in thedirections of mutually different gaze points (observation points) 111 to120. Each gaze point is an intersection point of an optical axis of thecorresponding image-capturing apparatus and the field, or a point in thevicinity of the intersection point. The gaze points 111 to 120, at whichthe image-capturing apparatuses 101 to 110 respectively are directed,may be set in advance as target positions. While the example isillustrated in FIG. 1 in which the image-capturing apparatuses 101 to110 are respectively pointed in the directions of the mutually differentgaze points 111 to 120, groups each including one or moreimage-capturing apparatuses (image-capturing apparatus group) may beconfigured to have mutually different gaze points.

In the first exemplary embodiment, a description will be provided offocus adjustment which is executed in each of the image-capturingapparatuses 101 to 110, in a case where the image-capturing apparatuses101 to 110 have the mutually different gaze points. While the number ofimage-capturing apparatuses is ten in the example illustrated in FIG. 1,the number is not limited to ten. While the example is illustrated inFIG. 1 in which the gaze points 111 to 120 are arranged on a half sideof the field 130, the arrangement is not limited thereto and the gazepoints 111 to 120 may be arranged on the whole surface of the field 130.Respective installation positions of the image-capturing apparatuses 101to 110 are determined by examining each stadium in advance andconsidering various conditions, such as the absence of obstruction ofview of the audience watching a game, an installation interval and aninstallation method for generating a virtual viewpoint image. For thisreason, the installation position of each image-capturing apparatus, adistance from each image-capturing apparatus to the field, and a heightfrom the field to the installation position of each image-capturingapparatus are different for each stadium. A focal length of a lens ofeach image-capturing apparatus is determined in consideration of, forexample, a distance from each image-capturing apparatus to a gaze point,a range in which the virtual viewpoint image is generated, and aresolution of an image of the subject included in the generated virtualviewpoint image.

FIG. 2 is a diagram schematically illustrating a configuration exampleof a system including the image-capturing apparatuses 101 to 110illustrated in FIG. 1, a control apparatus 200, and a hub 210 thatconnects the image-capturing apparatuses 101 to 110 to the controlapparatus 200.

The control apparatus 200 according to the present exemplary embodimenthas the function of controlling each of the image-capturing apparatuses101 to 110, and the function of generating the virtual viewpoint imagefrom a freely-selected viewpoint using images from a plurality ofviewpoints captured by the image-capturing apparatuses 101 to 110. Dataabout the images captured by the image-capturing apparatuses 101 to 110is transmitted through the hub 210 to a virtual viewpoint imagegeneration unit 230 in the control apparatus 200.

The virtual viewpoint image generation unit 230 generates a virtualviewpoint image 250 based on a position and photographing direction of avirtual image-capturing apparatus that are set by a virtual viewpointgeneration unit 240. While detailed description is omitted, the virtualviewpoint image generation unit 230 separates the images captured by therespective image-capturing apparatuses into the foreground and thebackground, generates a three-dimensional (3D) model from theforeground, and render the 3D model with colors viewed from the virtualimage-capturing apparatuses, thus generating the virtual viewpointimage. An algorithm to generate the virtual viewpoint image is notlimited thereto, and may be a method without generating the 3D model,such as a billboard method.

A user interface (UI) unit 260 is used for an operator to instruct eachimage-capturing apparatus to perform image-capturing and set the stateof each image-capturing apparatus. The UI unit 260 includes a displaydevice that displays an image among other items, and an operation unitthat acquires operation information in accordance with an operationperformed by the operator. The operator can instruct the image-capturingapparatus to start or end photographing, set a focal length, a shutterspeed, an aperture, sensitivity, and a focus position, through theoperation on the UI unit 260.

A control unit 220 controls photographing operation, a photographingdirection, a focal length, a shutter speed, an aperture, sensitivity,and a focus position with respect to each of the image-capturingapparatus 101 to 110, based on the instructions input by the operatorthrough the UI unit 260.

A system configuration is not limited to the example illustrated in FIG.2. The image-capturing apparatuses 101 to 110 may be directly connectedto the control unit 220 and/or the virtual viewpoint image generationunit 230 without interposing the hub 210. The image-capturingapparatuses 101 to 110 may be daisy-chained.

FIG. 3 is a diagram illustrating the functions of the control unit 220.FIG. 3 also illustrates the hub 210 and the UI unit 260.

The control unit 220 includes an image acquisition unit 321, a parameteracquisition unit 322, a command transmission unit 323, a depth of fieldcomputing unit 324, and a focus setting unit 325. The control unit 220further includes an area identification unit 326, a marker detectionunit 327, a marker registration unit 328, a position estimation unit329, a related information input unit 330, and a storage unit 331.

The image acquisition unit 321 acquires data about images (i.e., imagedata about moving images) consecutively captured by each of theimage-capturing apparatuses 101 to 110 through the hub 210, andtransmits the data to the area identification unit 326, the positionestimation unit 329, the marker detection unit 327, the storage unit331, and the UI unit 260.

As described above, the UI unit 260 displays the images from theimage-capturing apparatuses 101 to 110 on the display device, andacquires operation information in accordance with an operation performedby the operator through the operation unit. The operator can set, forexample, an angle of view, a photographing direction, or exposure basedon an aperture value for a desired image-capturing apparatus byoperating the operation unit while watching the images from theimage-capturing apparatuses 101 to 110, and adjust the desiredimage-capturing apparatus. The operator can visually check the focusstate of each image-capturing apparatus based on an image displayed onthe display device of the UI unit 260. Further, through operation on theUI unit 260, the operator can not only operate each of theimage-capturing apparatuses 101 to 110, but also operate eachimage-capturing apparatus of each group including one or moreimage-capturing apparatuses, or collectively operate all theimage-capturing apparatuses.

The parameter acquisition unit 322 acquires photographing parametersfrom each of the image-capturing apparatuses 101 to 110 through the hub210. The photographing parameters are current information with respectto each of the image-capturing apparatuses 101 to 110 about a zoomvalue, an aperture value, an exposure value, a focus value, and at whichposition in the screen the focus or exposure is adjusted. Thesephotographing parameters are also information indicating the state ofeach of the image-capturing apparatuses 101 to 110 at the time ofimage-capturing.

The marker detection unit 327 detects a specific marker pattern, whichwill be described below, from the captured images acquired by the imageacquisition unit 321, and identifies coordinates of the maker pattern inthe image. The marker pattern will be described in detail below.

One or more marker pattern images or feature amounts of the markerpattern images are registered (stored) in the marker registration unit328. The marker registration unit 328 stores the marker pattern imagesor information about the feature amounts in a hard disk or asemiconductor memory. The marker pattern images or the information aboutthe feature amounts registered in the marker registration unit 328 maybe prepared in advance, or acquired from, for example, an externalapparatus.

The storage unit 331 stores part or all of the images acquired by theimage acquisition unit 321. The storage unit 331 also stores informationabout the above-described photographing parameters of the correspondingimage-capturing apparatus that has captured the stored image inassociation with the stored image. The images and the photographingparameters stored in the storage unit 331 can be displayed on thedisplay device of the UI unit 260. In a case where the images capturedat the time of focus processing or the like executed in theimage-capturing apparatuses 101 to 110 and the information about thephotographing parameters are displayed on the display device of the UIunit 260, the operator can visually check the state of eachimage-capturing apparatus. That is, the operator can visually check thestate of the image captured by each image-capturing apparatus, and thestate of each image-capturing apparatus, such as a focus value, a focusstate at an in-focus position in the image, a zoom value, an aperturevalue, and an exposure value.

The depth of field computing unit 324 computes a depth of field at thein-focus position for each of the image-capturing apparatuses 101 to 110based on the photographing parameters acquired by the parameteracquisition unit 322 from the image-capturing apparatuses 101 to 110.

The related information input unit 330 acquires and holds informationabout the geometry of the stadium from the outside. The informationabout the geometry of the stadium is information including 3D geometricmodel data of the stadium and 3D coordinate information indicating theposition and size of the field 130 in the stadium. In the followingdescription, these pieces of information are collectively referred to asstadium-related information. The related information input unit 330holds the stadium-related information by storing the information in ahard disk or a semiconductor memory.

The position estimation unit 329 estimates, for each of theimage-capturing apparatuses 101 to 110, a relative position of thecorresponding image-capturing apparatus to the field 130, and a relativeorientation of each of the corresponding image-capturing apparatuses 101to 110 to the field 130. The position estimation unit 329 estimates therelative position and relative orientation of each image-capturingapparatus to the field 130 based on the stadium-related information heldby the related information input unit 330, and the captured imageacquired by the image acquisition unit 321 from each of theimage-capturing apparatuses 101 to 110.

The area identification unit 326 identifies an area on which eachimage-capturing apparatus should focus from the image acquired from theimage acquisition unit 321, the relative position of each of theimage-capturing apparatuses 101 to 110 to the field 130. The area onwhich each image-capturing apparatus should focus will be describedbelow. The relative position is estimated by the position estimationunit 329 described above.

The focus setting unit 325 sets the in-focus position within the angleof view of each image-capturing apparatus, i.e., within an area in whicheach image-capturing apparatus performs the image-capturing at afreely-selected position. The focus setting unit 325 can set, as thein-focus position, not only a position at the center of the angle ofview (center of area in which image-capturing apparatus performsimage-capturing) but also a position outside the center of the angle ofview. In the case of the present exemplary embodiment, the focus settingunit 325 sets, for an image-capturing apparatus, an area that has beenidentified by the area identification unit 326 as the area to be focusedon, as the in-focus position of the image-capturing apparatus in thefield 130 serving as the image-capturing target area. The area that isidentified as the area to be focused on will be described below. Thefocus setting unit 325 sets the area identified as the area to befocused on for each of the image-capturing apparatuses 101 to 110 as thein-focus position of the corresponding image-capturing apparatus.

The command transmission unit 323 generates various types of commandsrelated to photographing control, and transmits these commands to eachof the image-capturing apparatuses 101 to 110 through the hub 210. Inthe case of the present exemplary embodiment, the command transmissionunit 323 generates commands that provide instructions to, for example,start or end photographing, make a zoom setting, and set an aperturevalue, and a command related to focus processing. The commandtransmission unit 323 then transmits the commands to the image-capturingapparatuses 101 to 110. In the case of the present exemplary embodiment,the command related to the focus processing includes commands thatinstruct each image-capturing apparatus, for example, to start,interrupt, and end the focus processing, the in-focus position of eachimage-capturing apparatus on the screen. The actual focus processing isexecuted by the image-capturing apparatus.

In the case of the present exemplary embodiment, the focus adjustment ofeach of the image-capturing apparatuses 101 to 110 is executed in thearea identified by the area identification unit 326 as the area to befocused on. The area identified as the area to be focused on will bedescribed below.

First, an amount of blur produced by a lens of the image-capturingapparatus (camera) will be described.

FIG. 4 is a diagram illustrating a relationship between a distance fromthe image-capturing apparatus to the subject and an amount of blur witha certain aperture value and a focus value. The image-capturingapparatus is assumed to focus on an in-focus point 440 at the positionof the subject. The “in-focus” is assumed to be the same in meaning asfocus. At this time, the amount of blur sharply increases as the subjectapproaches the front side (image-capturing apparatus side) from thein-focus point 440. In contrast, the amount of blur increases little bylittle as the subject moves away from the in-focus point 440 toward thefar side (rear side). When an amount of blur that cannot be visuallyrecognized by humans is defined as an amount of blur 410 based on thedepth of field and the resolution of a human eye, a range, in which theimage-capturing apparatus can be regarded as being in an in-focus state,can be represented as an in-focus range 450. When a distance from thein-focus point 440 to a front end 451 of the in-focus range 450 is adistance A and a distance from the in-focus point 440 to a rear end 453of the in-focus range 450 is a sum of a distance B and a distance C, acenter 452 of the in-focus range 450 is on the rear side of the in-focuspoint 440. A sum of the distance A and the distance B is equal to thedistance C.

FIG. 5 is a diagram used for illustrating focus adjustment of animage-capturing apparatus 501, which is any one of the image-capturingapparatuses 101 to 110. In an example illustrated in FIG. 5, theimage-capturing apparatus 501 is assumed to focus on a gaze point 580(i.e., gaze point 580 corresponds to in-focus point 440). An in-focusrange 560 is a range in which the image-capturing apparatus 501 can beregarded as being in the in-focus state like the in-focus range 450illustrated in FIG. 4, and a front end 551 corresponds to the front end451 illustrated in FIG. 4 and a rear end 553 corresponds to the rear end453 illustrated in FIG. 4. A field 520 corresponds to the field 130illustrated in FIG. 1.

The example illustrated in FIG. 5 indicates a case in which the in-focusrange 560 is larger than the field 520. When the in-focus range 560 islarger than the field 520 as just described, the front end 551 and rearend 553 of the in-focus range 560 are outside the field 520. In such acase, an area in which the in-focus state is to be guaranteed and is tobe generated the virtual viewpoint image (in-focus guarantee area 570)is an area from a field end (front end 581) close to the image-capturingapparatus 501 to a field end (rear end 583) far from the image-capturingapparatus 501.

FIG. 6 is another diagram illustrating the focus adjustment of theimage-capturing apparatus 501, in a manner similar to the exampleillustrated in FIG. 5. Note that FIG. 6 illustrates a case in which thefield 520 is larger than the in-focus range 560, i.e., the in-focusrange 560 is smaller than the field 520. As illustrated in the examplein FIG. 6, when the field 520 is larger than the in-focus range 560, thefront end 551 and rear end 553 of the in-focus range 560 are inside thefield 520. In the case of the example illustrated in FIG. 6, thein-focus guarantee area 570 in which the in-focus state is to beguaranteed is the same as the area indicated by the in-focus range 560.That is, in the case of the example illustrated in FIG. 6, the front end581 of the in-focus guarantee area 570 coincides with the front end 551of the in-focus range 560, and the rear end 583 of the in-focusguarantee area 570 coincides with the rear end 553 of the in-focus range560.

The focus adjustment of the image-capturing apparatus 501 is to beperformed at least at the gaze point 580 and in an area in the vicinityof the gaze point 580 (referred to as gaze point area 530). In addition,in the case of the present exemplary embodiment, the focus adjustment ofthe image-capturing apparatus 501 is performed also in an area includingthe front end 581 of the in-focus guarantee area 570 (referred to asfront area 540) and an area including the rear end 583 of the in-focusguarantee area 570 (referred to as rear area 550). In the case of thepresent exemplary embodiment, the area identification unit 326identifies, for each of the image-capturing apparatuses 101 to 110, eachof the gaze point area 530, the front area 540, and the rear area 550 asthe area to be focused on in the image captured by the correspondingimage-capturing apparatus.

FIG. 7 is a diagram illustrating an example of a captured image displayscreen 700 in a case where the image captured by an image-capturingapparatus 501, which is any one of the image-capturing apparatuses 101to 110, is displayed on the display device of the UI unit 260. In a casewhere the gaze point area 530, the front area 540, and the rear area550, which have been identified by the area identification unit 326 asthe areas to be focused on, are displayed on the captured image displayscreen 700, the captured image display screen 700 displays a capturedimage as illustrated in FIG. 7. While the example of the captured imagedisplay screen 700 of one image-capturing apparatus 501 is illustratedin FIG. 7, the gaze point area, the front area, and the rear area aredifferent for each of the image-capturing apparatuses 101 to 110.

To perform the focus adjustment of the image-capturing apparatus 501 inthe gaze point area 530 or the like, a positional relationship in the 3Dcoordinate space of the field 130 among the position and orientation ofthe image-capturing apparatus 501, the gaze point 580, and the in-focusguarantee area 570 is to be determined.

FIG. 8 is a diagram illustrating a positional relationship in the 3Dcoordinate space between the respective positions and orientations ofthe image-capturing apparatuses 101 to 110 and the field 130. FIG. 8illustrates respective 3D coordinates (X1, Y1, Z1) to (X10, Y10, Z10) ofthe image-capturing apparatuses 101 to 110 with the origin (0, 0, 0) ofthe 3D coordinates at the center of the field 130. Assume that 3Dcoordinates at one end on an X-axis (long side direction) of the field130 are (Xf, 0, 0) and 3D coordinates at the other end on the X-axis ofthe field 130 are (−Xf, 0, 0) in FIG. 8. Similarly, assume that 3Dcoordinates at one end on a Y-axis (short side direction) of the field130 are (0, Yf, 0) and 3D coordinates at the other end on the Y-axis ofthe field 130 are (0, −Yf, 0). The 3D coordinates on the field 130 canbe acquired from the 3D coordinate information about the field 130included in the stadium-related information held by the relatedinformation input unit 330 described above. The actual size of the field130 can be determined from the 3D coordinate information about the field130 and the 3D geometric model data of the stadium. The determining ofthe actual size of the field 130 enables grasping of the distance fromeach of the image-capturing apparatuses 101 to 110 to the end of thefield 130 and the direction of each of the image-capturing apparatuses101 to 110. In the case of the present exemplary embodiment, theposition estimation unit 329 executes simple processing using thestadium-related information to estimate the distance from each of theimage-capturing apparatuses 101 to 110 to the end of the field 130 andthe direction of each of the image-capturing apparatuses 101 to 110.

The position estimation unit 329 also obtain mapping information betweena two-dimensional (2D) coordinate system of a captured image inaccordance with a zoom value of each of the image-capturing apparatuses101 to 110 and the above-described 3D coordinate system that is commonto the image-capturing apparatuses 101 to 110. The image-capturingapparatus 109 illustrated in FIGS. 8 and 9 among the image-capturingapparatuses 101 to 110 and a gaze point 119 of the image-capturingapparatus 109 are taken as an example. In this example, the positionestimation unit 329 acquires 2D coordinates (Xm, Ym) of a gaze point 930which are in the image captured by the image-capturing apparatus 109 (onsensor plane 940 of image-capturing apparatus 109) and correspond to 3Dcoordinates (Xn, Yn, Zn) of the gaze point 119 of the image-capturingapparatus 109.

In the present exemplary embodiment, a board on which the marker patternregistered in the marker registration unit 328 is printed is moved by amoving object, and the focus adjustment of the image-capturing apparatusis performed when the moving object enters any of the gaze point area,the front area, and the rear area. That is, the field used for a soccergame or the like has few features with which the focus adjustment of theimage-capturing apparatus can be performed, so that the focus adjustmentis performed using the marker pattern on the board assigned to themoving object.

FIGS. 10A to 10C are diagrams each illustrating a specific markerpattern example. A marker pattern illustrated in FIG. 10A is an exampleof a marker pattern for the gaze point to be used for the focusadjustment in the gaze point area. A marker pattern illustrated in FIG.10B is an example of a marker pattern for the front side to be used forthe focus adjustment in the front area. A marker pattern illustrated inFIG. 10C is an example of a marker pattern for the rear side to be usedfor the focus adjustment in the rear area. The number (types) of markerpatterns to be used may be more than that in this example or less thanthat in this example.

In FIG. 11, a person 1101 exemplifies the moving object to which theboard on which the marker patterns is printed or the like is assigned.FIG. 11 is a diagram illustrating the state of the person 1101 holding(carrying) the board on which the marker pattern is printed or the like.The moving object with the board on which the marker pattern is printedor the like is not limited to a person, and may be, for example, avehicle and a drone. Each of the moving objects moves at a predeterminedspeed on the field. Other than the case of the moving object holing theboard, the marker pattern may be projected on the field by, for example,a projector.

FIGS. 12A and 12B each illustrate an example of a route along which aplurality of moving objects, such as persons, for example, at leastthree moving objects 1201 to 1203 move on a field 1200 (field 130 inFIG. 1). For example, the moving object 1202 moves in the vicinity ofthe center area of the field 1200 as indicated by respective directionsof arrow in FIGS. 12A and 12B, and the moving objects 1201 and 1203 movein the vicinity of the respective ends of the field 1200 as indicated byrespective directions of arrow. Areas in which the moving objects 1201to 1203 move correspond to overlapping areas of the gaze point area, thefront area, and the rear area. Since the image-capturing apparatuses 101to 110 are installed in the stadium so as to surround the field 1200, inone embodiment, each of the moving objects 1201 to 1203 move in alongitudinal direction as illustrated in FIG. 12A, and thereafter moves,for example, in a transverse direction as illustrated in FIG. 12B.

FIG. 13 is a flowchart illustrating the flow of processing of thecontrol unit 220 when one of the image-capturing apparatuses 101 to 110executes the focus processing for the focus adjustment. Assume that theimage-capturing apparatuses 101 to 110 have already been installed inthe stadium, and have been set at a correct exposure.

First, as processing in step S1301, when supplied with data about one ormore specific maker patterns from, for example, an external apparatus,the control unit 220 registers the one or more marker patterns in themarker registration unit 328. The one or more specific marker patternsregistered in the marker registration unit 328 are used when a markerpattern corresponding to an image captured by the image-capturingapparatus is detected at a later process. The registration processingfor the marker patterns may be executed before the image-capturingapparatuses 101 to 110 are installed in the stadium. While the externalapparatus that supplies the marker patters to the marker registrationunit 328 is not particularly limited, it is assumed to be, for example,a personal computer.

Subsequently in step S1302, the command transmission unit 323 transmitsto the image-capturing apparatus a command (AF processing executionstart command) to cause the image-capturing apparatus to start autofocusprocessing (hereinafter referred to as AF processing). The transmissionof the AF processing execution start command in step S1302 is performedbefore operation of assigning the board on which the marker pattern isprinted or the like to the moving object and causing the moving objectto move on the field. The AF processing in the image-capturing apparatusat this time is executed such that the image-capturing apparatus focuseson turf on stadium, a white line, a goal, and audience seats, as analternative to the marker pattern. The focus adjustment of theimage-capturing apparatus at this time is not necessarily precise, andis to be at a level from which the marker pattern can be resolved if themarker pattern is placed on the field. The present exemplary embodimentassumes that the AF processing of the image-capturing apparatus isexecuted using any one of a phase difference detection method, acontrast difference detection method, and a method of the combinedadoption of the phase difference detection method and the contrastdifference detection method. The processing in step S1302 is executedindividually and in parallel in each of the image-capturing apparatuses101 to 110.

Subsequently in step S1303, the area identification unit 326 executesfocus area identification processing for identifying an area on whichthe image-capturing apparatus to be focus. In the case of the presentexemplary embodiment, the focus area identification processing includesprocessing for identifying the gaze point area, the front area, and therear area described above as the areas to be focused on with respect toeach of the image-capturing apparatuses 101 to 110.

When the processing until step S1303 ends, the processing proceeds tothe next step S1304. In step S1304, the image acquisition unit 321acquires images (i.e., images of moving image), which are consecutivelycaptured by the image-capturing apparatuses 101 to 110, of the movingobjects moving on the field in a state where the respective boards onwhich the respective marker patterns are printed or the like is held.Data about the images captured by the image-capturing apparatuses 101 to110 are then transmitted to the control unit 220 through the hub 210.

In step S1305, the marker detection unit 327 detects a marker patternthat matches with the marker pattern registered in the markerregistration unit 328 from the image acquired by the image acquisitionunit 321 from each of the image-capturing apparatuses 101 to 110.

Assume that the marker detection unit 327 detects the marker pattern forthe gaze point from the image captured by the image-capturing apparatuswhen the moving object holding the board with, for example, the markerpattern for the gaze point illustrated in FIG. 10A, enters the gazepoint area on the field. In this case, in step S1306, the commandtransmission unit 323 transmits the AF processing execution startcommand to the image-capturing apparatus that has captured the image.Thus, the image-capturing apparatus can execute the focus processing tofocus on the gaze point marker pattern on the board held by the movingobject.

In the next step in S1307 after the transmission of the AF processingexecution start command, the image acquisition unit 321 acquires thecaptured image from the image-capturing apparatus to which the AFprocessing execution start command has been transmitted from the commandtransmission unit 323. That is, the captured image that is acquired bythe image acquisition unit 321 from the image-capturing apparatus is animage in which the marker pattern for the gaze point is included in thegaze point area, and that has been captured by the image-capturingapparatus with the focus adjustment of focusing on the marker patternfor the gaze point. The captured image that has been acquired by theimage acquisition unit is transmitted to the storage unit 331 and storedin the storage unit 331. The image stored in the storage unit 331 isused for a user, such as an operator, to visually check whether thefocus state of the image-capturing apparatus is appropriate in the gazepoint area or for the control unit 220 to check this state based on, forexample, a contrast of the image.

Subsequently in step S1308, the marker detection unit 327 executesdetection processing of the marker pattern for the image captured by theimage-capturing apparatus when the moving object enters the front areaor the rear area in a manner similar to the gaze point area. Forexample, the marker detection unit 327 detects the marker pattern forthe front side from the image captured by the image-capturing apparatuswhen the moving object holding the board with, for example, the markerpattern for the front side illustrated in FIG. 10B, enters the frontarea on the field. Similarly, for example, the marker detection unit 327detects the marker pattern for the rear side from the image captured bythe image-capturing apparatus when the moving object holding the boardwith, for example, the marker pattern for the rear side illustrated inFIG. 10C, enters the rear area on the field.

Subsequently in step S1309, the image acquisition unit 321 acquires thecaptured image from the image-capturing apparatus. The captured imagethat is acquired by the image acquisition unit 321 from theimage-capturing apparatus is an image when the marker pattern for thefront side has been detected in the front area, or an image when themarker pattern for the rear side has been detected in the rear area. Thecaptured image that has been acquired by the image acquisition unit 321is transmitted to the storage unit 331 and stored in the storage unit331. The image stored in the storage unit 331 is used for the user tocheck whether the focus state of the image-capturing apparatus isappropriate in the front area or whether the focus state of theimage-capturing apparatus is appropriate in the rear area.

That is, in the case of the present exemplary embodiment, the capturedimages stored in step S1307 and step S1309 are used for the user tocheck whether the focus state of the image-capturing apparatus isappropriate in all of the gaze point area, the front area, and the reararea.

Timings at which the marker patterns on the respective boards held bythe respective moving objects enter the front area or the rear area aredifferent from one another. Thus, the marker detection unit 327 firstdetects the marker pattern for the front side that has entered the frontarea. Thereafter, the marker detection unit 327 detects the markerpattern for the rear side that has entered the rear area. Alternatively,the opposite order is possible. Further, there may be a case where themarker pattern for the rear side enters the rear area almostsimultaneously with the entry of the marker pattern for the front sidein the front area. In such a case, these marker patterns can be detectedsimultaneously.

FIG. 14 is a flowchart illustrating details of the focus areaidentification processing which is executed in step S1303 illustrated inFIG. 13.

First, in step S1401, the position estimation unit 329 acquires theimages acquired by the image acquisition unit 321 from theimage-capturing apparatuses 101 to 110 and the stadium-relatedinformation held by the related information input unit 330. The positionestimation unit 329 then estimates a positional relationship betweeneach of the image-capturing apparatuses 101 to 110 and the field 130,based on the image from each of the image-capturing apparatuses 101 to110 and the stadium-related information.

Subsequently in step S1402, the depth of field computing unit 324computes a depth of field of each of the image-capturing apparatuses 101to 110 from a current zoom value, a distance to the gaze point, and anaperture value of each of the image-capturing apparatuses 101 to 110.

Subsequently in step S1403, the area identification unit 326 identifiesthe gaze point area, the front area, and the rear area with respect toeach of the image-capturing apparatuses 101 to 110, based on thepositional relationship estimated in step S1401 and the depth of fieldacquired in step S1402. The control unit 220 then causes the displaydevice of the UI unit 260 to display the screen indicating the areasidentified in step S1403 as illustrated in FIG. 7. After step S1403, theprocessing proceeds to step S1304 illustrated in FIG. 13.

FIG. 15 is a flowchart illustrating details of the simple positionestimation processing executed in step S1306 illustrated in FIG. 13.

First, in step S1501, the position estimation unit 329 acquires one ormore images for each of the image-capturing apparatuses 101 to 110 fromthe corresponding images acquired by the image acquisition unit 321 fromthe image-capturing apparatuses 101 to 110.

Subsequently in step S1502, the position estimation unit 329 detectsnatural feature points from the corresponding images from theimage-capturing apparatuses 101 to 110. For example, the positionestimation unit 329 detects a gradient of luminance and a change incolor in the images as the natural feature points. A lot of naturalfeature points can be detected, for example, in portions of turf, agoal, a white line, a person, and audience seats in the images.

Subsequently in step S1503, the position estimation unit 329 executesmatching processing to match the natural feature points among therespective images captured by the image-capturing apparatuses 101 to110.

Subsequently in step S1504, the position estimation unit 329 estimates arelative position of each of the image-capturing apparatuses 101 to 110to the other image-capturing apparatuses. That is, a result of thematching processing for the natural feature points in step S1503 mayinclude an error. For this reason, in step S1504, the positionestimation unit 329 determines a relative position of theimage-capturing apparatuses 101 to 110 by minimizing the error.

Subsequently, in step S1505, the position estimation unit 329 acquiresthe stadium-related information from the related information input unit330.

Thereafter, in step S1506, the position estimation unit 329 associatesthe relative positions of the image-capturing apparatuses 101 to 110with the field based on the stadium-related information, and convertseach of the relative positions of the image-capturing apparatuses 101 to110 to an absolute position corresponding to the 3D coordinate space ofthe stadium. The position estimation unit 329 may use the 3D geometricmodel data to associate the relative positions of the image-capturingapparatuses 101 to 110 with the field, and may use a known 3D coordinatevalue on the field, if any, for the association. After step S1506, theprocessing proceeds to the processing in step S1304.

As described above, in the first exemplary embodiment, the focusadjustment of the image-capturing apparatus is processing to cause theimage-capturing apparatus to focus on the marker pattern printed on theboard held by the moving object that moves on the field. The focusadjustment of the image-capturing apparatus is performed when the markerpattern for the gaze point on the board held by the moving object entersthe gaze point area. Thus, according to the first exemplary embodiment,the focus adjustment of the image-capturing apparatuses can be executedin a short period of time even if there are a lot of gaze points.

A second exemplary embodiment of the disclosure will be described below.In the present exemplary embodiment, a description will be provided ofan example, in which not only the focus adjustment when eachimage-capturing apparatus has a different gaze point as illustrated inFIG. 1 is executed in a short period of time, but also calibrationoperation for acquiring the position and orientation of eachimage-capturing apparatus is executed in a short period of time.

FIG. 16 is a diagram illustrating a configuration example of a controlunit 1620 according to the second exemplary embodiment. In the controlunit 1620 illustrated in FIG. 16, the image acquisition unit 321 to thestorage unit 331 are equivalent to the respective units denoted by thecorresponding reference signs, and thus description thereof will beomitted. In the case of the second exemplary embodiment, the controlunit 1620 further includes a calibration computing unit 1632. Thecalibration computing unit 1632 executes calibration computingprocessing using the respective images acquired by the image acquisitionunit 321 from the image-capturing apparatuses 101 to 110 illustrated inFIG. 1.

A description will be provided below of processing of the control unit1620 according to the second exemplary embodiment to execute calibrationof the image-capturing apparatuses 101 to 110.

In the case where the virtual viewpoint image as described above isgenerated, the 3D geometry of the subject is restored from the imagesacquired by the image-capturing apparatuses 101 to 110. Thus, theposition and orientation of each of the image-capturing apparatuses 101to 110 is to be computed with high accuracy. The respective coordinatesof the image-capturing apparatuses 101 to 110 are to be associated witha coordinate system common to the image-capturing apparatuses 101 to110. Thus, the control unit 1620 according to the second exemplaryembodiment acquires the position and orientation of each of theimage-capturing apparatuses 101 to 110 with high accuracy and executesthe calibration computing processing (camera calibration computingprocessing) to perform the association of the coordinate system.

In the second exemplary embodiment, marker patterns for calibration areused to effectively execute calibration of the image-capturingapparatuses. The marker patterns for calibration are, for example,printed on the respective boards, which are carried by the respectivemoving objects (e.g., persons) so as to move in a common field of viewof the image-capturing apparatuses 101 to 110, i.e., on the field 130illustrated in FIG. 1. The control unit 1620 detects the respectivemarker patterns for calibration from the respective images acquired bycausing the image-capturing apparatuses 101 to 110 to performsynchronous image-capturing, and executes calibration of theimage-capturing apparatuses 101 to 110 based on the respective detectedmarker patterns for calibration.

FIG. 17 is a flowchart illustrating the flow of processing of thecontrol unit 1620 according to the second exemplary embodiment inexecuting the focus processing and the calibration computing processingof one image-capturing apparatus. In the flowchart illustrated in FIG.17, the operations from step S1303 to S1310 is similar to theabove-described operations in respective steps denoted by thecorresponding reference signs illustrated in FIG. 13, and descriptionthereof will be omitted. Also in this example, in a manner similar tothe above-described exemplary embodiment, a description will be providedon the assumption that the image-capturing apparatuses 101 to 110 havealready been installed in the stadium, and have been set at a correctexposure.

First, in step S1701, when supplied with data about a maker patternfrom, for example, the external apparatus, the control unit 1620registers the marker pattern in the marker registration unit 328. Whilethe marker patterns registered in the marker registration unit 328 areused to detect the marker patterns from the respective captured imagesfrom the image-capturing apparatuses in a later process, the markerpatterns according to the second exemplary embodiment include thepatterns for calibration. The registration processing of the markerpatterns may be executed before the image-capturing apparatuses 101 to110 are installed in the stadium.

FIGS. 18A to 18F are diagrams each illustrating a specific markerpattern example according to the second exemplary embodiment. Markerpatterns illustrated in FIGS. 18A and 18B are the marker patterns forthe gaze point used in the gaze point area as in the first exemplaryembodiment, and also used as the marker patterns for calibration. Themarker pattern illustrated in FIG. 18A is configured such that a markeridentification (ID) 1801 for calibration is superimposed on (added to)the marker pattern illustrated in FIG. 10A. The marker patternillustrated in FIG. 18B is configured such that a marker ID 1802 forcalibration is superimposed on (added to) the marker pattern illustratedin FIG. 10A. Similarly, a marker pattern illustrated in FIG. 18C isconfigured such that a marker ID 1803 for calibration is superimposed onthe marker pattern for the front side described above, and a markerpattern illustrated in FIG. 18D is configured such that a marker ID 1804for calibration is superimposed on the marker pattern for the frontside. Similarly, a marker pattern illustrated in FIG. 18E is configuredsuch that a marker ID 1805 for calibration is superimposed on the markerpattern for the rear side described above, and a marker patternillustrated in FIG. 18F is configured such that a marker ID 1806 forcalibration is superimposed on the marker pattern for the rear side. Thenumber (types) of marker patterns to be used may be more than that inthis example or less than that in this example.

Subsequently, in step S1702, the command transmission unit 323 transmitsto the image-capturing apparatuses 101 to 110 to instruct theimage-capturing apparatuses 101 to 110 to start synchronousimage-capturing. Accordingly, the image-capturing apparatuses 101 to 110perform synchronous image-capturing. The reason that the image-capturingapparatuses are caused to perform the synchronous image-capturing instep S1702 is to execute the matching processing using the markerpattern with the identical marker ID and the natural feature pointsamong images that are time-synchronous in the calibration computingprocessing in a later process. The operations in subsequent steps S1302to S1309 is similar to those in the above-described flowchart in FIG.13, and after step S1309, the processing proceeds to step S1711.

Subsequently, in step S1711, the command transmission unit 323 transmitsto the image-capturing apparatuses 101 to 110 commands that instruct theimage-capturing apparatuses 101 to 110 to keep the synchronousimage-capturing and start to capture images to be used for thecalibration computing. Accordingly, the image-capturing apparatuses 101to 110 perform image-capturing of time-synchronous images (images usedfor calibration computing).

Subsequently in step S1712, the image acquisition unit 321 acquires thetime-synchronous images that are synchronously captured by theimage-capturing apparatus 101 to 110. The storage unit 331 stores thetime-synchronous images in association with the respectiveimage-capturing apparatuses.

Subsequently in step S1713, the calibration computing unit 1632 detectsan image with the marker pattern for calibration from each of thetime-synchronous images stored in the storage unit 331. As describedabove, the marker pattern for calibration includes the marker ID. Thus,by detecting the marker ID from the image stored in the storage unit331, the calibration computing unit 1632 can identify the image as theimage for the calibration computing. The calibration computing unit 1632executes the matching processing using the marker pattern with theidentical marker ID and the natural feature points among thetime-synchronous images, and executes the calibration computingprocessing by detecting the marker pattern for calibration. The methodfor executing the calibration computing with respect to a plurality ofimage-capturing apparatuses using the specific marker pattern such asthe marker pattern for calibration is a known technique, and thusdescription thereof will be omitted.

As described above, the second exemplary embodiment enables execution ofthe focus adjustment of the image-capturing apparatuses 101 to 110 as inthe first exemplary embodiment, and also enables execution of thecalibration of the image-capturing apparatuses 101 to 110. The secondexemplary embodiment achieves reduction in time in the focus adjustmentas well as the calibration processing by providing commonality betweenthe marker pattern for the focus adjustment and the marker pattern forthe calibration computing, and causing the image-capturing apparatuses101 to 110 to perform the synchronous image-capturing.

FIG. 19 is a diagram illustrating a hardware configuration example of aninformation processing apparatus 1900 as an application example of theabove-described exemplary embodiments. The information processingapparatus 1900 is, for example, a personal computer. The informationprocessing apparatus 1900 illustrated in FIG. 19 includes a centralprocessing unit (CPU) 1911, a read-only memory (ROM) 1912, arandom-access memory (RAM) 1913, an auxiliary storage device 1914, adisplay unit 1915, and operation unit 1916, a communication interface(I/F) 1917, and a bus 1918. The bus 1918 connects the units of theinformation processing apparatus 1900 and transmits information to theunits.

The CPU 1911 controls the whole of the information processing apparatus1900 using a computer program and data stored in the ROM 1912 and theRAM 1913. The CPU 1911 can execute processing of the control unit 220,the virtual viewpoint image generation unit 230, the virtual viewpointgeneration unit 240, and the UI unit 260 described above using theprogram and data according to the exemplary embodiments described above.The information processing apparatus 1900 includes one or more dedicatedhardware devices different from the CPU 1911, and the dedicated hardwaredevice may execute at least part of the processing executed by the CPU1911. Examples of the dedicated hardware device include anapplication-specific integrated circuit (ASIC), a field-programmablegate array (FPGA), and a digital signal processor (DSP). The ROM 1912stores a program or the like that does not need to be changed. The RAM1913 temporarily stores a program and data supplied from the auxiliarystorage device 1914, and data supplied from the outside through thecommunication I/F 1917. The auxiliary storage device 1914 includes, forexample, a hard disk drive, and stores a variety of data, such as imagedata and audio data.

The display unit 1915 includes, for example, a display device, such as aliquid crystal display and an a light emitting diode (LED) display, anddisplays, for example, a graphical user interface (GUI) for the user,such as the operator, to operate the information processing apparatus1900 and images captured by the image-capturing apparatuses on a screen.The operation unit 1916 includes, for example, a keyboard, a mouse, ajoy stick, a touch panel, and inputs a variety of instructions inresponse to the user's operation to the CPU 1911. The display unit 1915and the operation unit 1916 are included in the UI unit 260 describedabove.

While the display unit 1915 and the operation unit 1916 are incorporatedin the information processing apparatus 1900 in this example, at leastone of the display unit 1915 and the operation unit 1916 may be providedas a separate device outside the information processing apparatus 1900.In such a case, the CPU 1911 may operate as a display control unit thatcontrols the display unit 1915 and as an operation control unit thatcontrols the operation unit 1916.

The communication I/F 1917 is used for communication with an externalapparatus outside the information processing apparatus 1900. Forexample, in a case where the information processing apparatus 1900 haswired connection to the external apparatus, a cable for communication isconnected to the communication I/F 1917. In a case where the informationprocessing apparatus 1900 has the function of wirelessly communicatingwith the external apparatus, the communication I/F 1917 is provided withan antenna. The information processing apparatus 1900 can communicatewith the image-capturing apparatuses 101 to 110 through thecommunication I/F 1917 and the hub 210 described above. With thisconfiguration, the CPU 1911 can control, for example, operation andsettings of the image-capturing apparatuses 101 to 110.

Processing executed by, for example, the control unit 220 and thevirtual viewpoint image generation unit 230, among the units of thecontrol apparatus 200 described above, may be processing to whichartificial intelligence (AI) is applied. For example, a trained modeltrained through machine learning may be used in substitution for theseunits. In such a case, a plurality of combinations of input data andoutput data to and from these units is prepared as training data, andthe trained model, which has acquired knowledge through machine-learningfrom the combinations and outputs data with respect to input data as aresult based on the acquired knowledge, is generated. The trained modelcan be configured using a neural network model, for example. The trainedmodel, which serves as a program for executing the processing in anequivalent manner to the above-described units, executes the processingof the respective units in collaboration with the CPU, the GPU, and thelike. Such a trained model can be updated or the like as necessary everytime a certain amount of data is processed.

A program that achieves one or more functions in the control processingaccording to the exemplary embodiments of the disclosure can be suppliedto a system or an apparatus through a network or a storage media, andcan be achieved by being loaded and executed by one or more processorsin the system or a computer of the apparatus.

The exemplary embodiments described above are mere concrete examples ofimplementing the disclosure, and the technical scope of the disclosureshould not be interpreted in a limited manner by the exemplaryembodiments. That is, the disclosure can be implemented in various modeswithout departing from the technical idea or the principal features ofthe disclosure.

Embodiment(s) of the disclosure can also be realized by a computer of asystem or apparatus that reads out and executes computer executableinstructions (e.g., one or more programs) recorded on a storage medium(which may also be referred to more fully as a ‘non-transitorycomputer-readable storage medium’) to perform the functions of one ormore of the above-described embodiment(s) and/or that includes one ormore circuits (e.g., application specific integrated circuit (ASIC)) forperforming the functions of one or more of the above-describedembodiment(s), and by a method performed by the computer of the systemor apparatus by, for example, reading out and executing the computerexecutable instructions from the storage medium to perform the functionsof one or more of the above-described embodiment(s) and/or controllingthe one or more circuits to perform the functions of one or more of theabove-described embodiment(s). The computer may comprise one or moreprocessors (e.g., central processing unit (CPU), micro processing unit(MPU)) and may include a network of separate computers or separateprocessors to read out and execute the computer executable instructions.The computer executable instructions may be provided to the computer,for example, from a network or the storage medium. The storage mediummay include, for example, one or more of a hard disk, a random-accessmemory (RAM), a read only memory (ROM), a storage of distributedcomputing systems, an optical disk (such as a compact disc (CD), digitalversatile disc (DVD), or Blu-ray Disc (BD)™), a flash memory device, amemory card, and the like.

The present disclosure enables execution of the focus adjustment of animage-capturing apparatus in a short period of time.

While the disclosure has been described with reference to exemplaryembodiments, it is to be understood that the disclosure is not limitedto the disclosed exemplary embodiments. The scope of the followingclaims is to be accorded the broadest interpretation so as to encompassall such modifications and equivalent structures and functions.

This application claims the benefit of Japanese Patent Application No.2019-146278, filed Aug. 8, 2019, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. A control apparatus configured to control animage-capturing apparatus, the control apparatus comprising: one or morememories storing instructions; and one or more processors executing theinstructions to: specify an area to be focused by the image-capturingapparatus in an image-capturing target area captured by a plurality ofimage-capturing apparatuses including the image-capturing apparatus;detect a specific marker pattern from an image acquired theimage-capturing apparatus, the specific marker pattern being moved inthe image-capturing target area captured by the plurality ofimage-capturing apparatuses; and execute focus adjustment of theimage-capturing apparatus using the specific marker pattern, based onthe detection of the specific marker pattern from an image of the areathat is to be focused by the image-capturing apparatus and has beenspecified in the acquired image.
 2. The control apparatus according toclaim 1, further comprising a storage device configured to store theacquired image and information about a state of the image-capturingapparatus when the focus adjustment is executed.
 3. The controlapparatus according to claim 2, wherein the acquired image includes thespecific marker pattern in the area that is to be focused by theimage-capturing apparatus.
 4. The control apparatus according to claim2, wherein the storage device stores the acquired image including thespecific marker pattern in the area that is to be focused by theimage-capturing apparatus and another image including another markerpattern in another area that is to be focused by the image-capturingapparatus.
 5. The control apparatus according to claim 1, wherein thefocus adjustment is executed for each of the plurality ofimage-capturing apparatuses arranged such that each group including oneor more of the image-capturing apparatuses has a gaze point.
 6. Thecontrol apparatus according to claim 1, wherein the area to be focusedby the image-capturing apparatus is identified based on an imagecaptured by the image-capturing apparatus and a relative positionalrelationship between the image-capturing apparatus and theimage-capturing target area.
 7. The control apparatus according to claim6, wherein the area to be focused by the image-capturing apparatus; isspecified based on an in-focus range that is determined by a distancefrom the image-capturing apparatus to a subject and an amount of blurproduced by a lens included in the image-capturing apparatus, and on anin-focus guarantee area in which an in-focus state of theimage-capturing apparatus is to be guaranteed for generation of an imageusing the captured image.
 8. The control apparatus according to claim 1,wherein the specific marker pattern is assigned to a moving object thatmoves in the image-capturing target area, and wherein the specificmarker pattern is detected from an image of the moving object thatappears in an image captured in the image-capturing target area by theimage-capturing apparatus.
 9. The control apparatus according to claim8, wherein the specific marker pattern is detected from an image of oneof one or more moving objects appearing in the image captured in theimage-capturing target area by the image-capturing apparatus.
 10. Thecontrol apparatus according to claim 1, wherein the area to be focusedby the image-capturing apparatus includes a plurality of areas, andwherein the focus adjustment of the image-capturing apparatus isexecuted in response to the detection of the specific marker patternfrom an image of one of the plurality of areas that is to be focused bythe image-capturing apparatus and has been identified in the acquiredimage.
 11. The control apparatus according to claim 1, wherein thespecific marker pattern includes a pattern for calibration to acquire aposition and orientation of the image-capturing apparatus.
 12. Thecontrol apparatus according to claim 1, wherein the one or moreprocessors further execute the instructions to perform focus adjustmentof the image-capturing apparatus using a still object in the area thatis to be focused by the image-capturing apparatus, before the detectionof the specific marker pattern.
 13. The control apparatus according toclaim 1, wherein the focus adjustment is executed for each of theplurality of image-capturing apparatuses, each of the plurality ofimage-capturing apparatuses being pointed to one of plurality of gazepoints.
 14. A control method comprising: specifying an area to befocused by an image-capturing apparatus in an image-capturing targetarea captured by a plurality of image-capturing apparatuses includingthe image-capturing apparatus; detecting a specific marker pattern froman image acquired by the image-capturing apparatus, the specific markerpattern being moved in the image-capturing target area captured by theplurality of image-capturing apparatuses; and executing focus adjustmentof the image-capturing apparatus using the specific marker pattern,based on the detection of the specific marker pattern from an image ofthe specified area to be focused by the image-capturing apparatus in theacquired image.
 15. The control method according to claim 14, furthercomprising storing the acquired image and information about a state ofthe image-capturing apparatus when the focus adjustment is executed. 16.The control method according to claim 14, further comprising executingthe focus adjustment for each of the plurality of image-capturingapparatuses arranged such that each group including one or more of theimage-capturing apparatuses has a gaze point.
 17. The control methodaccording to claim 14, wherein the area to be focused by theimage-capturing apparatus is specified based on an image captured by theimage-capturing apparatus and a relative positional relationship betweenthe image-capturing apparatus and the image-capturing target area. 18.The control method according to claim 14, wherein the specific markerpattern is assigned to a moving object that moves in the image-capturingtarget area, and wherein the specific marker pattern is detected from animage of the moving object that appears in an image captured in theimage-capturing target area by the image-capturing apparatus.
 19. Anon-transitory computer-readable recording medium configured to store aprogram that causes a computer to execute a control method comprising:specifying an area to be focused by an image-capturing apparatus in animage-capturing target area captured by a plurality of image-capturingapparatuses including the image-capturing apparatus; detecting aspecific marker pattern from an image acquired by the image-capturingapparatus, the specific marker pattern being moved in theimage-capturing target area captured by the plurality of image-capturingapparatuses; and executing focus adjustment of the image-capturingapparatus using the specific marker pattern, based on the detection ofthe specific marker pattern from an image of the specified area to befocused by the image-capturing apparatus in the acquired image.
 20. Thenon-transitory computer-readable recording medium according to claim 19,further comprising storing the acquired image and information about astate of the image-capturing apparatus when the focus adjustment isexecuted.
 21. The non-transitory computer-readable recording mediumaccording to claim 19, further comprising specifying the area to befocused by the image-capturing apparatus based on an image captured bythe image-capturing apparatus and a relative positional relationshipbetween the image-capturing apparatus and the image-capturing targetarea.